{
  "cells": [
    {
      "cell_type": "raw",
      "id": "afaf8039",
      "metadata": {
        "vscode": {
          "languageId": "raw"
        }
      },
      "source": [
        "---\n",
        "sidebar_label: OpenAI\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e49f1e0d",
      "metadata": {},
      "source": [
        "# ChatOpenAI\n",
        "\n",
        "[OpenAI](https://en.wikipedia.org/wiki/OpenAI) is an artificial intelligence (AI) research laboratory.\n",
        "\n",
        "This guide will help you getting started with ChatOpenAI [chat models](/docs/concepts/chat_models). For detailed documentation of all ChatOpenAI features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html).\n",
        "\n",
        "## Overview\n",
        "### Integration details\n",
        "\n",
        "| Class | Package | Local | Serializable | [PY support](https://python.langchain.com/docs/integrations/chat/openai) | Package downloads | Package latest |\n",
        "| :--- | :--- | :---: | :---: |  :---: | :---: | :---: |\n",
        "| [ChatOpenAI](https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) | [`@langchain/openai`](https://www.npmjs.com/package/@langchain/openai) | ❌ | ✅ | ✅ | ![NPM - Downloads](https://img.shields.io/npm/dm/@langchain/openai?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/@langchain/openai?style=flat-square&label=%20&) |\n",
        "\n",
        "### Model features\n",
        "\n",
        "See the links in the table headers below for guides on how to use specific features.\n",
        "\n",
        "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
        "| :---: | :---: | :---: | :---: |  :---: | :---: | :---: | :---: | :---: |\n",
        "| ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | \n",
        "\n",
        "## Setup\n",
        "\n",
        "To access OpenAI chat models you'll need to create an OpenAI account, get an API key, and install the `@langchain/openai` integration package.\n",
        "\n",
        "### Credentials\n",
        "\n",
        "Head to [OpenAI's website](https://platform.openai.com/) to sign up for OpenAI and generate an API key. Once you've done this set the `OPENAI_API_KEY` environment variable:\n",
        "\n",
        "```bash\n",
        "export OPENAI_API_KEY=\"your-api-key\"\n",
        "```\n",
        "\n",
        "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n",
        "\n",
        "```bash\n",
        "# export LANGSMITH_TRACING=\"true\"\n",
        "# export LANGSMITH_API_KEY=\"your-api-key\"\n",
        "```\n",
        "\n",
        "### Installation\n",
        "\n",
        "The LangChain `ChatOpenAI` integration lives in the `@langchain/openai` package:\n",
        "\n",
        "```{=mdx}\n",
        "\n",
        "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n",
        "import Npm2Yarn from \"@theme/Npm2Yarn\";\n",
        "\n",
        "<IntegrationInstallTooltip></IntegrationInstallTooltip>\n",
        "\n",
        "<Npm2Yarn>\n",
        "  @langchain/openai @langchain/core\n",
        "</Npm2Yarn>\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a38cde65-254d-4219-a441-068766c0d4b5",
      "metadata": {},
      "source": [
        "## Instantiation\n",
        "\n",
        "Now we can instantiate our model object and generate chat completions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\" \n",
        "\n",
        "const llm = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  temperature: 0,\n",
        "  // other params...\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2b4f3e15",
      "metadata": {},
      "source": [
        "## Invocation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "62e0dbc3",
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "AIMessage {\n",
            "  \"id\": \"chatcmpl-ADItECqSPuuEuBHHPjeCkh9wIO1H5\",\n",
            "  \"content\": \"J'adore la programmation.\",\n",
            "  \"additional_kwargs\": {},\n",
            "  \"response_metadata\": {\n",
            "    \"tokenUsage\": {\n",
            "      \"completionTokens\": 5,\n",
            "      \"promptTokens\": 31,\n",
            "      \"totalTokens\": 36\n",
            "    },\n",
            "    \"finish_reason\": \"stop\",\n",
            "    \"system_fingerprint\": \"fp_5796ac6771\"\n",
            "  },\n",
            "  \"tool_calls\": [],\n",
            "  \"invalid_tool_calls\": [],\n",
            "  \"usage_metadata\": {\n",
            "    \"input_tokens\": 31,\n",
            "    \"output_tokens\": 5,\n",
            "    \"total_tokens\": 36\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "const aiMsg = await llm.invoke([\n",
        "  {\n",
        "    role: \"system\",\n",
        "    content: \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n",
        "  },\n",
        "  {\n",
        "    role: \"user\",\n",
        "    content: \"I love programming.\"\n",
        "  },\n",
        "])\n",
        "aiMsg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "J'adore la programmation.\n"
          ]
        }
      ],
      "source": [
        "console.log(aiMsg.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8",
      "metadata": {},
      "source": [
        "## Chaining\n",
        "\n",
        "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "AIMessage {\n",
            "  \"id\": \"chatcmpl-ADItFaWFNqkSjSmlxeGk6HxcBHzVN\",\n",
            "  \"content\": \"Ich liebe Programmieren.\",\n",
            "  \"additional_kwargs\": {},\n",
            "  \"response_metadata\": {\n",
            "    \"tokenUsage\": {\n",
            "      \"completionTokens\": 5,\n",
            "      \"promptTokens\": 26,\n",
            "      \"totalTokens\": 31\n",
            "    },\n",
            "    \"finish_reason\": \"stop\",\n",
            "    \"system_fingerprint\": \"fp_5796ac6771\"\n",
            "  },\n",
            "  \"tool_calls\": [],\n",
            "  \"invalid_tool_calls\": [],\n",
            "  \"usage_metadata\": {\n",
            "    \"input_tokens\": 26,\n",
            "    \"output_tokens\": 5,\n",
            "    \"total_tokens\": 31\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "import { ChatPromptTemplate } from \"@langchain/core/prompts\"\n",
        "\n",
        "const prompt = ChatPromptTemplate.fromMessages(\n",
        "  [\n",
        "    [\n",
        "      \"system\",\n",
        "      \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n",
        "    ],\n",
        "    [\"human\", \"{input}\"],\n",
        "  ]\n",
        ")\n",
        "\n",
        "const chain = prompt.pipe(llm);\n",
        "await chain.invoke(\n",
        "  {\n",
        "    input_language: \"English\",\n",
        "    output_language: \"German\",\n",
        "    input: \"I love programming.\",\n",
        "  }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "06ffc86b",
      "metadata": {},
      "source": [
        "## Custom URLs\n",
        "\n",
        "You can customize the base URL the SDK sends requests to by passing a `configuration` parameter like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "19a092b9",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llmWithCustomURL = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  temperature: 0.9,\n",
        "  configuration: {\n",
        "    baseURL: \"https://your_custom_url.com\",\n",
        "  },\n",
        "});\n",
        "\n",
        "await llmWithCustomURL.invoke(\"Hi there!\");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "20b60ccb",
      "metadata": {},
      "source": [
        "The `configuration` field also accepts other `ClientOptions` parameters accepted by the official SDK.\n",
        "\n",
        "If you are hosting on Azure OpenAI, see the [dedicated page instead](/docs/integrations/chat/azure).\n",
        "\n",
        "## Custom headers\n",
        "\n",
        "You can specify custom headers in the same `configuration` field:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cd612609",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llmWithCustomHeaders = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  temperature: 0.9,\n",
        "  configuration: {\n",
        "    defaultHeaders: {\n",
        "      \"Authorization\": `Bearer SOME_CUSTOM_VALUE`,\n",
        "    },\n",
        "  },\n",
        "});\n",
        "\n",
        "await llmWithCustomHeaders.invoke(\"Hi there!\");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7af61d1d",
      "metadata": {},
      "source": [
        "## Disabling streaming usage metadata\n",
        "\n",
        "Some proxies or third-party providers present largely the same API interface as OpenAI, but don't support the more recently added `stream_options` parameter to return streaming usage. You can use `ChatOpenAI` to access these providers by disabling streaming usage like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0ff40bd7",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llmWithoutStreamUsage = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  temperature: 0.9,\n",
        "  streamUsage: false,\n",
        "  configuration: {\n",
        "    baseURL: \"https://proxy.com\",\n",
        "  },\n",
        "});\n",
        "\n",
        "await llmWithoutStreamUsage.invoke(\"Hi there!\");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "013b6300",
      "metadata": {},
      "source": [
        "## Calling fine-tuned models\n",
        "\n",
        "You can call fine-tuned OpenAI models by passing in your corresponding `modelName` parameter.\n",
        "\n",
        "This generally takes the form of `ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}`. For example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7448f6a9",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const fineTunedLlm = new ChatOpenAI({\n",
        "  temperature: 0.9,\n",
        "  model: \"ft:gpt-3.5-turbo-0613:{ORG_NAME}::{MODEL_ID}\",\n",
        "});\n",
        "\n",
        "await fineTunedLlm.invoke(\"Hi there!\");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a2270901",
      "metadata": {},
      "source": [
        "## Generation metadata\n",
        "\n",
        "If you need additional information like logprobs or token usage, these will be returned directly in the `.invoke` response within the `response_metadata` field on the message.\n",
        "\n",
        "```{=mdx}\n",
        "\n",
        ":::tip\n",
        "Requires `@langchain/core` version >=0.1.48.\n",
        ":::\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "2b675330",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  content: [\n",
            "    {\n",
            "      token: 'Hello',\n",
            "      logprob: -0.0004740447,\n",
            "      bytes: [ 72, 101, 108, 108, 111 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: '!',\n",
            "      logprob: -0.00004334534,\n",
            "      bytes: [ 33 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: ' How',\n",
            "      logprob: -0.000030113732,\n",
            "      bytes: [ 32, 72, 111, 119 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: ' can',\n",
            "      logprob: -0.0004797665,\n",
            "      bytes: [ 32, 99, 97, 110 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: ' I',\n",
            "      logprob: -7.89631e-7,\n",
            "      bytes: [ 32, 73 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: ' assist',\n",
            "      logprob: -0.114006,\n",
            "      bytes: [\n",
            "         32,  97, 115,\n",
            "        115, 105, 115,\n",
            "        116\n",
            "      ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: ' you',\n",
            "      logprob: -4.3202e-7,\n",
            "      bytes: [ 32, 121, 111, 117 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: ' today',\n",
            "      logprob: -0.00004501419,\n",
            "      bytes: [ 32, 116, 111, 100, 97, 121 ],\n",
            "      top_logprobs: []\n",
            "    },\n",
            "    {\n",
            "      token: '?',\n",
            "      logprob: -0.000010206721,\n",
            "      bytes: [ 63 ],\n",
            "      top_logprobs: []\n",
            "    }\n",
            "  ],\n",
            "  refusal: null\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "// See https://cookbook.openai.com/examples/using_logprobs for details\n",
        "const llmWithLogprobs = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "  logprobs: true,\n",
        "  // topLogprobs: 5,\n",
        "});\n",
        "\n",
        "const responseMessageWithLogprobs = await llmWithLogprobs.invoke(\"Hi there!\");\n",
        "console.dir(responseMessageWithLogprobs.response_metadata.logprobs, { depth: null });"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",
      "metadata": {},
      "source": [
        "## Tool calling\n",
        "\n",
        "Tool calling with OpenAI models works in a similar to [other models](/docs/how_to/tool_calling). Additionally, the following guides have some information especially relevant to OpenAI:\n",
        "\n",
        "- [How to: disable parallel tool calling](/docs/how_to/tool_calling_parallel/)\n",
        "- [How to: force a tool call](/docs/how_to/tool_choice/)\n",
        "- [How to: bind model-specific tool formats to a model](/docs/how_to/tool_calling#binding-model-specific-formats-advanced)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4e6b50cc",
      "metadata": {},
      "source": [
        "### Custom Tools\n",
        "\n",
        "[Custom tools](https://platform.openai.com/docs/guides/function-calling#custom-tools) support tools with arbitrary string inputs. They can be particularly useful when you expect your string arguments to be long or complex.\n",
        "\n",
        "If you use a model that supports custom tools, you can use the `ChatOpenAI` class and the `customTool` function to create a custom tool."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "66589441",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI, customTool } from \"@langchain/openai\";\n",
        "import { createReactAgent } from \"@langchain/langgraph/prebuilt\";\n",
        "\n",
        "const codeTool = customTool(\n",
        "  async (input) => {\n",
        "    // ... Add code to execute the input\n",
        "    return \"Code executed successfully\";\n",
        "  },\n",
        "  {\n",
        "    name: \"execute_code\",\n",
        "    description: \"Execute a code snippet\",\n",
        "    format: { type: \"text\" },\n",
        "  }\n",
        ");\n",
        "\n",
        "const model = new ChatOpenAI({ model: \"gpt-5\" });\n",
        "\n",
        "const agent = createReactAgent({\n",
        "  llm: model,\n",
        "  tools: [codeTool],\n",
        "});\n",
        "\n",
        "const result = await agent.invoke(\"Use the tool to calculate 3^3\");\n",
        "console.log(result);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eba4fc5d",
      "metadata": {},
      "source": [
        "<details>\n",
        "<summary>Context-free grammars</summary>\n",
        "\n",
        "OpenAI supports the specification of a [context-free grammar](https://platform.openai.com/docs/guides/function-calling#context-free-grammars) for custom tool inputs in `lark` or `regex` format. See [OpenAI docs](https://platform.openai.com/docs/guides/function-calling#context-free-grammars) for details. The `format` parameter can be passed into `customTool` as shown below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d0a54d72",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI, customTool } from \"@langchain/openai\";\n",
        "import { createReactAgent } from \"@langchain/langgraph/prebuilt\";\n",
        "\n",
        "const MATH_GRAMMAR = `\n",
        "start: expr\n",
        "expr: term (SP ADD SP term)* -> add\n",
        "| term\n",
        "term: factor (SP MUL SP factor)* -> mul\n",
        "| factor\n",
        "factor: INT\n",
        "SP: \\\" \\\"\n",
        "ADD: \\\"+\\\"\n",
        "MUL: \\\"*\\\"\n",
        "%import common.INT\n",
        "`;\n",
        "\n",
        "const doMath = customTool(\n",
        "  async (input) => {\n",
        "    // ... Add code to parse and execute the input\n",
        "    return \"27\";\n",
        "  },\n",
        "  {\n",
        "    name: \"do_math\",\n",
        "    description: \"Evaluate a math expression\",\n",
        "    format: { type: \"grammar\", grammar: MATH_GRAMMAR },\n",
        "  }\n",
        ");\n",
        "\n",
        "const model = new ChatOpenAI({ model: \"gpt-5\" });\n",
        "\n",
        "const agent = createReactAgent({\n",
        "  llm: model,\n",
        "  tools: [doMath],\n",
        "});\n",
        "\n",
        "const result = await agent.invoke(\"Use the tool to calculate 3^3\");\n",
        "console.log(result);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "73cacf58",
      "metadata": {},
      "source": [
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3392390e",
      "metadata": {},
      "source": [
        "## ``strict: true``\n",
        "\n",
        "As of Aug 6, 2024, OpenAI supports a `strict` argument when calling tools that will enforce that the tool argument schema is respected by the model. See more here: https://platform.openai.com/docs/guides/function-calling.\n",
        "\n",
        "```{=mdx}\n",
        "\n",
        ":::info Requires ``@langchain/openai >= 0.2.6``\n",
        "\n",
        "**Note**: If ``strict: true`` the tool definition will also be validated, and a subset of JSON schema are accepted. Crucially, schema cannot have optional args (those with default values). Read the full docs on what types of schema are supported here: https://platform.openai.com/docs/guides/structured-outputs/supported-schemas. \n",
        ":::\n",
        "\n",
        "\n",
        "```\n",
        "\n",
        "Here's an example with tool calling. Passing an extra `strict: true` argument to `.bindTools` will pass the param through to all tool definitions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "90f0d465",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "  {\n",
            "    name: 'get_current_weather',\n",
            "    args: { location: 'current' },\n",
            "    type: 'tool_call',\n",
            "    id: 'call_hVFyYNRwc6CoTgr9AQFQVjm9'\n",
            "  }\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "import { tool } from \"@langchain/core/tools\";\n",
        "import { z } from \"zod\";\n",
        "\n",
        "const weatherTool = tool((_) => \"no-op\", {\n",
        "  name: \"get_current_weather\",\n",
        "  description: \"Get the current weather\",\n",
        "  schema: z.object({\n",
        "    location: z.string(),\n",
        "  }),\n",
        "})\n",
        "\n",
        "const llmWithStrictTrue = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "}).bindTools([weatherTool], {\n",
        "  strict: true,\n",
        "  tool_choice: weatherTool.name,\n",
        "});\n",
        "\n",
        "// Although the question is not about the weather, it will call the tool with the correct arguments\n",
        "// because we passed `tool_choice` and `strict: true`.\n",
        "const strictTrueResult = await llmWithStrictTrue.invoke(\"What is 127862 times 12898 divided by 2?\");\n",
        "\n",
        "console.dir(strictTrueResult.tool_calls, { depth: null });"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6c46a668",
      "metadata": {},
      "source": [
        "If you only want to apply this parameter to a select number of tools, you can also pass OpenAI formatted tool schemas directly:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "e2da9ead",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "  {\n",
            "    name: 'get_current_weather',\n",
            "    args: { location: 'London' },\n",
            "    type: 'tool_call',\n",
            "    id: 'call_EOSejtax8aYtqpchY8n8O82l'\n",
            "  }\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "import { zodToJsonSchema } from \"zod-to-json-schema\";\n",
        "\n",
        "const toolSchema = {\n",
        "  type: \"function\",\n",
        "  function: {\n",
        "    name: \"get_current_weather\",\n",
        "    description: \"Get the current weather\",\n",
        "    strict: true,\n",
        "    parameters: zodToJsonSchema(\n",
        "      z.object({\n",
        "        location: z.string(),\n",
        "      })\n",
        "    ),\n",
        "  },\n",
        "};\n",
        "\n",
        "const llmWithStrictTrueTools = new ChatOpenAI({\n",
        "  model: \"gpt-4o\",\n",
        "}).bindTools([toolSchema], {\n",
        "  strict: true,\n",
        "});\n",
        "\n",
        "const weatherToolResult = await llmWithStrictTrueTools.invoke([{\n",
        "  role: \"user\",\n",
        "  content: \"What is the current weather in London?\"\n",
        "}])\n",
        "\n",
        "weatherToolResult.tool_calls;"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "045668fe",
      "metadata": {},
      "source": [
        "## Structured output\n",
        "\n",
        "We can also pass `strict: true` to the [`.withStructuredOutput()`](https://js.langchain.com/docs/how_to/structured_output/#the-.withstructuredoutput-method). Here's an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "8e8171a5",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{ traits: [ `6'5\" tall`, 'love fruit' ] }\n"
          ]
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const traitSchema = z.object({\n",
        "  traits: z.array(z.string()).describe(\"A list of traits contained in the input\"),\n",
        "});\n",
        "\n",
        "const structuredLlm = new ChatOpenAI({\n",
        "  model: \"gpt-4o-mini\",\n",
        "}).withStructuredOutput(traitSchema, {\n",
        "  name: \"extract_traits\",\n",
        "  strict: true,\n",
        "});\n",
        "\n",
        "await structuredLlm.invoke([{\n",
        "  role: \"user\",\n",
        "  content: `I am 6'5\" tall and love fruit.`\n",
        "}]);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7139f50e",
      "metadata": {},
      "source": [
        "## Responses API\n",
        "\n",
        ":::caution Compatibility\n",
        "\n",
        "The below points apply to `@langchain/openai>=0.4.5-rc.0`. Please see here for a [guide on upgrading](/docs/how_to/installation/#installing-integration-packages).\n",
        "\n",
        ":::\n",
        "\n",
        "OpenAI supports a [Responses](https://platform.openai.com/docs/guides/responses-vs-chat-completions) API that is oriented toward building [agentic](/docs/concepts/agents/) applications. It includes a suite of [built-in tools](https://platform.openai.com/docs/guides/tools?api-mode=responses), including web and file search. It also supports management of [conversation state](https://platform.openai.com/docs/guides/conversation-state?api-mode=responses), allowing you to continue a conversational thread without explicitly passing in previous messages.\n",
        "\n",
        "`ChatOpenAI` will route to the Responses API if one of these features is used. You can also specify `useResponsesApi: true` when instantiating `ChatOpenAI`.\n",
        "\n",
        "### Built-in tools\n",
        "\n",
        "Equipping `ChatOpenAI` with built-in tools will ground its responses with outside information, such as via context in files or the web. The [AIMessage](/docs/concepts/messages/#aimessage) generated from the model will include information about the built-in tool invocation.\n",
        "\n",
        "#### Web search\n",
        "\n",
        "To trigger a web search, pass `{\"type\": \"web_search_preview\"}` to the model as you would another tool.\n",
        "\n",
        ":::tip\n",
        "\n",
        "You can also pass built-in tools as invocation params:\n",
        "\n",
        "```ts\n",
        "llm.invoke(\"...\", { tools: [{ type: \"web_search_preview\" }] });\n",
        "```\n",
        "\n",
        ":::\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1b01f09a",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llm = new ChatOpenAI({ model: \"gpt-4o-mini\" }).bindTools([\n",
        "  { type: \"web_search_preview\" },\n",
        "]);\n",
        "\n",
        "await llm.invoke(\"What was a positive news story from today?\");\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e62bae97",
      "metadata": {},
      "source": [
        "Note that the response includes structured [content blocks](/docs/concepts/messages/#content-1) that include both the text of the response and OpenAI [annotations](https://platform.openai.com/docs/guides/tools-web-search?api-mode=responses#output-and-citations) citing its sources. The output message will also contain information from any tool invocations.\n",
        "\n",
        "#### File search\n",
        "\n",
        "To trigger a file search, pass a [file search tool](https://platform.openai.com/docs/guides/tools-file-search) to the model as you would another tool. You will need to populate an OpenAI-managed vector store and include the vector store ID in the tool definition. See [OpenAI documentation](https://platform.openai.com/docs/guides/tools-file-search) for more details.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b70c671c",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llm = new ChatOpenAI({ model: \"gpt-4o-mini\" }).bindTools([\n",
        "  { type: \"file_search\", vector_store_ids: [\"vs...\"] },\n",
        "]);\n",
        "\n",
        "await llm.invoke(\"Is deep research by OpenAI?\");\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e2a8378c",
      "metadata": {},
      "source": [
        "As with [web search](#web-search), the response will include content blocks with citations. It will also include information from the built-in tool invocations.\n",
        "\n",
        "#### Computer Use\n",
        "\n",
        "ChatOpenAI supports the `computer-use-preview` model, which is a specialized model for the built-in computer use tool. To enable, pass a [computer use tool](https://platform.openai.com/docs/guides/tools-computer-use) as you would pass another tool. \n",
        "\n",
        "Currently tool outputs for computer use are present in `AIMessage.additional_kwargs.tool_outputs`. To reply to the computer use tool call, you need to set `additional_kwargs.type: \"computer_call_output\"` while creating a corresponding `ToolMessage`.\n",
        "\n",
        "See [OpenAI documentation](https://platform.openai.com/docs/guides/tools-computer-use) for more details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "176eb3ea",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { AIMessage, ToolMessage } from \"@langchain/core/messages\";\n",
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "import * as fs from \"node:fs/promises\";\n",
        "\n",
        "const findComputerCall = (message: AIMessage) => {\n",
        "  const toolOutputs = message.additional_kwargs.tool_outputs as\n",
        "    | { type: \"computer_call\"; call_id: string; action: { type: string } }[]\n",
        "    | undefined;\n",
        "\n",
        "  return toolOutputs?.find((toolOutput) => toolOutput.type === \"computer_call\");\n",
        "};\n",
        "\n",
        "const llm = new ChatOpenAI({ model: \"computer-use-preview\" })\n",
        "  .bindTools([\n",
        "    {\n",
        "      type: \"computer-preview\",\n",
        "      display_width: 1024,\n",
        "      display_height: 768,\n",
        "      environment: \"browser\",\n",
        "    },\n",
        "  ])\n",
        "  .bind({ truncation: \"auto\" });\n",
        "\n",
        "let message = await llm.invoke(\"Check the latest OpenAI news on bing.com.\");\n",
        "const computerCall = findComputerCall(message);\n",
        "\n",
        "if (computerCall) {\n",
        "  // Act on a computer call action\n",
        "  const screenshot = await fs.readFile(\"./screenshot.png\", {\n",
        "    encoding: \"base64\",\n",
        "  });\n",
        "\n",
        "  message = await llm.invoke(\n",
        "    [\n",
        "      new ToolMessage({\n",
        "        additional_kwargs: { type: \"computer_call_output\" },\n",
        "        tool_call_id: computerCall.call_id,\n",
        "        content: [\n",
        "          {\n",
        "            type: \"computer_screenshot\",\n",
        "            image_url: `data:image/png;base64,${screenshot}`,\n",
        "          },\n",
        "        ],\n",
        "      }),\n",
        "    ],\n",
        "    { previous_response_id: message.response_metadata[\"id\"] }\n",
        "  );\n",
        "}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2918529b",
      "metadata": {},
      "source": [
        "#### Code interpreter\n",
        "\n",
        "ChatOpenAI allows you to use the built-in [code interpreter tool](https://platform.openai.com/docs/guides/tools-code-interpreter) to support the sandboxed generation and execution of code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c9185748",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llm = new ChatOpenAI({\n",
        "  model: \"o4-mini\",\n",
        "  useResponsesApi: true,\n",
        "});\n",
        "\n",
        "const llmWithTools = llm.bindToools([\n",
        "  {\n",
        "    type: \"code_interpreter\",\n",
        "    // Creates a new container\n",
        "    container: { type: \"auto\" }\n",
        "  },\n",
        "]);\n",
        "\n",
        "const response = await llmWithTools.invoke(\n",
        "  \"Write and run code to answer the question: what is 3^3?\"\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1771a7ab",
      "metadata": {},
      "source": [
        "Note that the above command creates a new [container](https://platform.openai.com/docs/guides/tools-code-interpreter#containers). We can re-use containers across calls by specifying an existing container ID."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4eeea048",
      "metadata": {},
      "outputs": [],
      "source": [
        "const tool_outputs: Record<string, any>[] = response.additional_kwargs.tool_outputs\n",
        "const container_id = tool_outputs[0].container_id\n",
        "\n",
        "const llmWithTools = llm.bindTools([\n",
        "  {\n",
        "    type: \"code_interpreter\",\n",
        "    // Re-uses container from the last call\n",
        "    container: container_id,\n",
        "  },\n",
        "]);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0e8f980b",
      "metadata": {},
      "source": [
        "#### Remote MCP\n",
        "\n",
        "ChatOpenAI supports the built-in [remote MCP tool](https://platform.openai.com/docs/guides/tools-remote-mcp) that allows for model-generated calls to MCP servers to happen on OpenAI servers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b6a2fe41",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llm = new ChatOpenAI({\n",
        "    model: \"o4-mini\",\n",
        "    useResponsesApi: true,\n",
        "});\n",
        "\n",
        "const llmWithMcp = llm.bindTools([\n",
        "    {\n",
        "        type: \"mcp\",\n",
        "        server_label: \"deepwiki\",\n",
        "        server_url: \"https://mcp.deepwiki.com/mcp\",\n",
        "        require_approval: \"never\"\n",
        "    }\n",
        "]);\n",
        "\n",
        "const response = await llmWithMcp.invoke(\n",
        "    \"What transport protocols does the 2025-03-26 version of the MCP spec (modelcontextprotocol/modelcontextprotocol) support?\"\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d7bb9788",
      "metadata": {},
      "source": [
        "```{=mdx}\n",
        "<details>\n",
        "<summary>MCP Approvals</summary>\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When instructed, OpenAI will request approval before making calls to a remote MCP server.\n",
        "\n",
        "In the above command, we instructed the model to never require approval. We can also configure the model to always request approval, or to always request approval for specific tools:\n",
        "\n",
        "```typescript\n",
        "...\n",
        "const llmWithMcp = llm.bindTools([\n",
        "  {\n",
        "    type: \"mcp\",\n",
        "    server_label: \"deepwiki\",\n",
        "    server_url: \"https://mcp.deepwiki.com/mcp\",\n",
        "    require_approval: {\n",
        "      always: {\n",
        "        tool_names: [\"read_wiki_structure\"],\n",
        "      },\n",
        "    },\n",
        "  },\n",
        "]);\n",
        "const response = await llmWithMcp.invoke(\n",
        "    \"What transport protocols does the 2025-03-26 version of the MCP spec (modelcontextprotocol/modelcontextprotocol) support?\"\n",
        ");\n",
        "```\n",
        "\n",
        "With this configuration, responses can contain tool outputs typed as `mcp_approval_request`. To submit approvals for an approval request, you can structure it into a content block in a followup message:\n",
        "\n",
        "```typescript\n",
        "const approvals = [];\n",
        "if (Array.isArray(response.additional_kwargs.tool_outputs)) {\n",
        "  for (const content of response.additional_kwargs.tool_outputs) {\n",
        "    if (content.type === \"mcp_approval_request\") {\n",
        "      approvals.push({\n",
        "        type: \"mcp_approval_response\",\n",
        "        approval_request_id: content.id,\n",
        "        approve: true,\n",
        "      });\n",
        "    }\n",
        "  }\n",
        "}\n",
        "\n",
        "const nextResponse = await model.invoke(\n",
        "  [\n",
        "    response,\n",
        "    new HumanMessage({ content: approvals }),\n",
        "  ],\n",
        ");\n",
        "```\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5d0a44ae",
      "metadata": {},
      "source": [
        "```{=mdx}\n",
        "</details>\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a8695925",
      "metadata": {},
      "source": [
        "#### Image Generation\n",
        "\n",
        "ChatOpenAI allows you to bring the built-in [image generation tool](https://platform.openai.com/docs/guides/tools-image-generation) to create images as apart of multi-turn conversations through the responses API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "82ff1906",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llm = new ChatOpenAI({\n",
        "  model: \"gpt-4.1\",\n",
        "  useResponsesApi: true,\n",
        "});\n",
        "\n",
        "const llmWithImageGeneration = llm.bindTools([\n",
        "  {\n",
        "    type: \"image_generation\",\n",
        "    quality: \"low\",\n",
        "  }\n",
        "]);\n",
        "\n",
        "const response = await llmWithImageGeneration.invoke(\n",
        "  \"Draw a random short word in green font.\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0194ec1f",
      "metadata": {},
      "source": [
        "### Reasoning models\n",
        "\n",
        "```{=mdx}\n",
        ":::caution Compatibility\n",
        "\n",
        "The below points apply to `@langchain/openai>=0.4.0`. Please see here for a [guide on upgrading](/docs/how_to/installation/#installing-integration-packages).\n",
        "\n",
        ":::\n",
        "```\n",
        "\n",
        "When using reasoning models like `o1`, the default method for `withStructuredOutput` is OpenAI's built-in method for structured output (equivalent to passing `method: \"jsonSchema\"` as an option into `withStructuredOutput`). JSON schema mostly works the same as other models, but with one important caveat: when defining schema, `z.optional()` is not respected, and you should instead use `z.nullable()`.\n",
        "\n",
        "Here's an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "d2a04807",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{ color: 'No color mentioned' }\n"
          ]
        }
      ],
      "source": [
        "import { z } from \"zod\";\n",
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "// Will not work\n",
        "const reasoningModelSchemaOptional = z.object({\n",
        "  color: z.optional(z.string()).describe(\"A color mentioned in the input\"),\n",
        "});\n",
        "\n",
        "const reasoningModelOptionalSchema = new ChatOpenAI({\n",
        "  model: \"o1\",\n",
        "}).withStructuredOutput(reasoningModelSchemaOptional, {\n",
        "  name: \"extract_color\",\n",
        "});\n",
        "\n",
        "await reasoningModelOptionalSchema.invoke([{\n",
        "  role: \"user\",\n",
        "  content: `I am 6'5\" tall and love fruit.`\n",
        "}]);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "69854ed4",
      "metadata": {},
      "source": [
        "And here's an example with `z.nullable()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "5f4bb1bc",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{ color: null }\n"
          ]
        }
      ],
      "source": [
        "import { z } from \"zod\";\n",
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "// Will not work\n",
        "const reasoningModelSchemaNullable = z.object({\n",
        "  color: z.nullable(z.string()).describe(\"A color mentioned in the input\"),\n",
        "});\n",
        "\n",
        "const reasoningModelNullableSchema = new ChatOpenAI({\n",
        "  model: \"o1\",\n",
        "}).withStructuredOutput(reasoningModelSchemaNullable, {\n",
        "  name: \"extract_color\",\n",
        "});\n",
        "\n",
        "await reasoningModelNullableSchema.invoke([{\n",
        "  role: \"user\",\n",
        "  content: `I am 6'5\" tall and love fruit.`\n",
        "}]);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "af20e756",
      "metadata": {},
      "source": [
        "## Prompt caching\n",
        "\n",
        "Newer OpenAI models will automatically [cache parts of your prompt](https://openai.com/index/api-prompt-caching/) if your inputs are above a certain size (1024 tokens at the time of writing) in order to reduce costs for use-cases that require long context.\n",
        "\n",
        "**Note:** The number of tokens cached for a given query is not yet standardized in `AIMessage.usage_metadata`, and is instead contained in the `AIMessage.response_metadata` field.\n",
        "\n",
        "Here's an example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "cb4e4fd0",
      "metadata": {},
      "outputs": [],
      "source": [
        "// @lc-docs-hide-cell\n",
        "\n",
        "const CACHED_TEXT = `## Components\n",
        "\n",
        "LangChain provides standard, extendable interfaces and external integrations for various components useful for building with LLMs.\n",
        "Some components LangChain implements, some components we rely on third-party integrations for, and others are a mix.\n",
        "\n",
        "### Chat models\n",
        "\n",
        "<span data-heading-keywords=\"chat model,chat models\"></span>\n",
        "\n",
        "Language models that use a sequence of messages as inputs and return chat messages as outputs (as opposed to using plain text).\n",
        "These are generally newer models (older models are generally \\`LLMs\\`, see below).\n",
        "Chat models support the assignment of distinct roles to conversation messages, helping to distinguish messages from the AI, users, and instructions such as system messages.\n",
        "\n",
        "Although the underlying models are messages in, message out, the LangChain wrappers also allow these models to take a string as input.\n",
        "This gives them the same interface as LLMs (and simpler to use).\n",
        "When a string is passed in as input, it will be converted to a \\`HumanMessage\\` under the hood before being passed to the underlying model.\n",
        "\n",
        "LangChain does not host any Chat Models, rather we rely on third party integrations.\n",
        "\n",
        "We have some standardized parameters when constructing ChatModels:\n",
        "\n",
        "- \\`model\\`: the name of the model\n",
        "\n",
        "Chat Models also accept other parameters that are specific to that integration.\n",
        "\n",
        ":::important\n",
        "Some chat models have been fine-tuned for **tool calling** and provide a dedicated API for it.\n",
        "Generally, such models are better at tool calling than non-fine-tuned models, and are recommended for use cases that require tool calling.\n",
        "Please see the [tool calling section](/docs/concepts/tool_calling) for more information.\n",
        ":::\n",
        "\n",
        "For specifics on how to use chat models, see the [relevant how-to guides here](/docs/how_to/#chat-models).\n",
        "\n",
        "#### Multimodality\n",
        "\n",
        "Some chat models are multimodal, accepting images, audio and even video as inputs.\n",
        "These are still less common, meaning model providers haven't standardized on the \"best\" way to define the API.\n",
        "Multimodal outputs are even less common. As such, we've kept our multimodal abstractions fairly light weight\n",
        "and plan to further solidify the multimodal APIs and interaction patterns as the field matures.\n",
        "\n",
        "In LangChain, most chat models that support multimodal inputs also accept those values in OpenAI's content blocks format.\n",
        "So far this is restricted to image inputs. For models like Gemini which support video and other bytes input, the APIs also support the native, model-specific representations.\n",
        "\n",
        "For specifics on how to use multimodal models, see the [relevant how-to guides here](/docs/how_to/#multimodal).\n",
        "\n",
        "### LLMs\n",
        "\n",
        "<span data-heading-keywords=\"llm,llms\"></span>\n",
        "\n",
        ":::caution\n",
        "Pure text-in/text-out LLMs tend to be older or lower-level. Many popular models are best used as [chat completion models](/docs/concepts/chat_models),\n",
        "even for non-chat use cases.\n",
        "\n",
        "You are probably looking for [the section above instead](/docs/concepts/chat_models).\n",
        ":::\n",
        "\n",
        "Language models that takes a string as input and returns a string.\n",
        "These are traditionally older models (newer models generally are [Chat Models](/docs/concepts/chat_models), see above).\n",
        "\n",
        "Although the underlying models are string in, string out, the LangChain wrappers also allow these models to take messages as input.\n",
        "This gives them the same interface as [Chat Models](/docs/concepts/chat_models).\n",
        "When messages are passed in as input, they will be formatted into a string under the hood before being passed to the underlying model.\n",
        "\n",
        "LangChain does not host any LLMs, rather we rely on third party integrations.\n",
        "\n",
        "For specifics on how to use LLMs, see the [relevant how-to guides here](/docs/how_to/#llms).\n",
        "\n",
        "### Message types\n",
        "\n",
        "Some language models take an array of messages as input and return a message.\n",
        "There are a few different types of messages.\n",
        "All messages have a \\`role\\`, \\`content\\`, and \\`response_metadata\\` property.\n",
        "\n",
        "The \\`role\\` describes WHO is saying the message.\n",
        "LangChain has different message classes for different roles.\n",
        "\n",
        "The \\`content\\` property describes the content of the message.\n",
        "This can be a few different things:\n",
        "\n",
        "- A string (most models deal this type of content)\n",
        "- A List of objects (this is used for multi-modal input, where the object contains information about that input type and that input location)\n",
        "\n",
        "#### HumanMessage\n",
        "\n",
        "This represents a message from the user.\n",
        "\n",
        "#### AIMessage\n",
        "\n",
        "This represents a message from the model. In addition to the \\`content\\` property, these messages also have:\n",
        "\n",
        "**\\`response_metadata\\`**\n",
        "\n",
        "The \\`response_metadata\\` property contains additional metadata about the response. The data here is often specific to each model provider.\n",
        "This is where information like log-probs and token usage may be stored.\n",
        "\n",
        "**\\`tool_calls\\`**\n",
        "\n",
        "These represent a decision from an language model to call a tool. They are included as part of an \\`AIMessage\\` output.\n",
        "They can be accessed from there with the \\`.tool_calls\\` property.\n",
        "\n",
        "This property returns a list of \\`ToolCall\\`s. A \\`ToolCall\\` is an object with the following arguments:\n",
        "\n",
        "- \\`name\\`: The name of the tool that should be called.\n",
        "- \\`args\\`: The arguments to that tool.\n",
        "- \\`id\\`: The id of that tool call.\n",
        "\n",
        "#### SystemMessage\n",
        "\n",
        "This represents a system message, which tells the model how to behave. Not every model provider supports this.\n",
        "\n",
        "#### ToolMessage\n",
        "\n",
        "This represents the result of a tool call. In addition to \\`role\\` and \\`content\\`, this message has:\n",
        "\n",
        "- a \\`tool_call_id\\` field which conveys the id of the call to the tool that was called to produce this result.\n",
        "- an \\`artifact\\` field which can be used to pass along arbitrary artifacts of the tool execution which are useful to track but which should not be sent to the model.\n",
        "\n",
        "#### (Legacy) FunctionMessage\n",
        "\n",
        "This is a legacy message type, corresponding to OpenAI's legacy function-calling API. \\`ToolMessage\\` should be used instead to correspond to the updated tool-calling API.\n",
        "\n",
        "This represents the result of a function call. In addition to \\`role\\` and \\`content\\`, this message has a \\`name\\` parameter which conveys the name of the function that was called to produce this result.\n",
        "\n",
        "### Prompt templates\n",
        "\n",
        "<span data-heading-keywords=\"prompt,prompttemplate,chatprompttemplate\"></span>\n",
        "\n",
        "Prompt templates help to translate user input and parameters into instructions for a language model.\n",
        "This can be used to guide a model's response, helping it understand the context and generate relevant and coherent language-based output.\n",
        "\n",
        "Prompt Templates take as input an object, where each key represents a variable in the prompt template to fill in.\n",
        "\n",
        "Prompt Templates output a PromptValue. This PromptValue can be passed to an LLM or a ChatModel, and can also be cast to a string or an array of messages.\n",
        "The reason this PromptValue exists is to make it easy to switch between strings and messages.\n",
        "\n",
        "There are a few different types of prompt templates:\n",
        "\n",
        "#### String PromptTemplates\n",
        "\n",
        "These prompt templates are used to format a single string, and generally are used for simpler inputs.\n",
        "For example, a common way to construct and use a PromptTemplate is as follows:\n",
        "\n",
        "\\`\\`\\`typescript\n",
        "import { PromptTemplate } from \"@langchain/core/prompts\";\n",
        "\n",
        "const promptTemplate = PromptTemplate.fromTemplate(\n",
        "  \"Tell me a joke about {topic}\"\n",
        ");\n",
        "\n",
        "await promptTemplate.invoke({ topic: \"cats\" });\n",
        "\\`\\`\\`\n",
        "\n",
        "#### ChatPromptTemplates\n",
        "\n",
        "These prompt templates are used to format an array of messages. These \"templates\" consist of an array of templates themselves.\n",
        "For example, a common way to construct and use a ChatPromptTemplate is as follows:\n",
        "\n",
        "\\`\\`\\`typescript\n",
        "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
        "\n",
        "const promptTemplate = ChatPromptTemplate.fromMessages([\n",
        "  [\"system\", \"You are a helpful assistant\"],\n",
        "  [\"user\", \"Tell me a joke about {topic}\"],\n",
        "]);\n",
        "\n",
        "await promptTemplate.invoke({ topic: \"cats\" });\n",
        "\\`\\`\\`\n",
        "\n",
        "In the above example, this ChatPromptTemplate will construct two messages when called.\n",
        "The first is a system message, that has no variables to format.\n",
        "The second is a HumanMessage, and will be formatted by the \\`topic\\` variable the user passes in.\n",
        "\n",
        "#### MessagesPlaceholder\n",
        "\n",
        "<span data-heading-keywords=\"messagesplaceholder\"></span>\n",
        "\n",
        "This prompt template is responsible for adding an array of messages in a particular place.\n",
        "In the above ChatPromptTemplate, we saw how we could format two messages, each one a string.\n",
        "But what if we wanted the user to pass in an array of messages that we would slot into a particular spot?\n",
        "This is how you use MessagesPlaceholder.\n",
        "\n",
        "\\`\\`\\`typescript\n",
        "import {\n",
        "  ChatPromptTemplate,\n",
        "  MessagesPlaceholder,\n",
        "} from \"@langchain/core/prompts\";\n",
        "import { HumanMessage } from \"@langchain/core/messages\";\n",
        "\n",
        "const promptTemplate = ChatPromptTemplate.fromMessages([\n",
        "  [\"system\", \"You are a helpful assistant\"],\n",
        "  new MessagesPlaceholder(\"msgs\"),\n",
        "]);\n",
        "\n",
        "promptTemplate.invoke({ msgs: [new HumanMessage({ content: \"hi!\" })] });\n",
        "\\`\\`\\`\n",
        "\n",
        "This will produce an array of two messages, the first one being a system message, and the second one being the HumanMessage we passed in.\n",
        "If we had passed in 5 messages, then it would have produced 6 messages in total (the system message plus the 5 passed in).\n",
        "This is useful for letting an array of messages be slotted into a particular spot.\n",
        "\n",
        "An alternative way to accomplish the same thing without using the \\`MessagesPlaceholder\\` class explicitly is:\n",
        "\n",
        "\\`\\`\\`typescript\n",
        "const promptTemplate = ChatPromptTemplate.fromMessages([\n",
        "  [\"system\", \"You are a helpful assistant\"],\n",
        "  [\"placeholder\", \"{msgs}\"], // <-- This is the changed part\n",
        "]);\n",
        "\\`\\`\\`\n",
        "\n",
        "For specifics on how to use prompt templates, see the [relevant how-to guides here](/docs/how_to/#prompt-templates).\n",
        "\n",
        "### Example Selectors\n",
        "\n",
        "One common prompting technique for achieving better performance is to include examples as part of the prompt.\n",
        "This gives the language model concrete examples of how it should behave.\n",
        "Sometimes these examples are hardcoded into the prompt, but for more advanced situations it may be nice to dynamically select them.\n",
        "Example Selectors are classes responsible for selecting and then formatting examples into prompts.\n",
        "\n",
        "For specifics on how to use example selectors, see the [relevant how-to guides here](/docs/how_to/#example-selectors).\n",
        "\n",
        "### Output parsers\n",
        "\n",
        "<span data-heading-keywords=\"output parser\"></span>\n",
        "\n",
        ":::note\n",
        "\n",
        "The information here refers to parsers that take a text output from a model try to parse it into a more structured representation.\n",
        "More and more models are supporting function (or tool) calling, which handles this automatically.\n",
        "It is recommended to use function/tool calling rather than output parsing.\n",
        "See documentation for that [here](/docs/concepts/tool_calling).\n",
        "\n",
        ":::\n",
        "\n",
        "Responsible for taking the output of a model and transforming it to a more suitable format for downstream tasks.\n",
        "Useful when you are using LLMs to generate structured data, or to normalize output from chat models and LLMs.\n",
        "\n",
        "There are two main methods an output parser must implement:\n",
        "\n",
        "- \"Get format instructions\": A method which returns a string containing instructions for how the output of a language model should be formatted.\n",
        "- \"Parse\": A method which takes in a string (assumed to be the response from a language model) and parses it into some structure.\n",
        "\n",
        "And then one optional one:\n",
        "\n",
        "- \"Parse with prompt\": A method which takes in a string (assumed to be the response from a language model) and a prompt (assumed to be the prompt that generated such a response) and parses it into some structure. The prompt is largely provided in the event the OutputParser wants to retry or fix the output in some way, and needs information from the prompt to do so.\n",
        "\n",
        "Output parsers accept a string or \\`BaseMessage\\` as input and can return an arbitrary type.\n",
        "\n",
        "LangChain has many different types of output parsers. This is a list of output parsers LangChain supports. The table below has various pieces of information:\n",
        "\n",
        "**Name**: The name of the output parser\n",
        "\n",
        "**Supports Streaming**: Whether the output parser supports streaming.\n",
        "\n",
        "**Input Type**: Expected input type. Most output parsers work on both strings and messages, but some (like OpenAI Functions) need a message with specific arguments.\n",
        "\n",
        "**Output Type**: The output type of the object returned by the parser.\n",
        "\n",
        "**Description**: Our commentary on this output parser and when to use it.\n",
        "\n",
        "The current date is ${new Date().toISOString()}`;\n",
        "\n",
        "// Noop statement to hide output\n",
        "void 0;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "7a43595c",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "USAGE: {\n",
            "  prompt_tokens: 2624,\n",
            "  completion_tokens: 263,\n",
            "  total_tokens: 2887,\n",
            "  prompt_tokens_details: { cached_tokens: 0 },\n",
            "  completion_tokens_details: { reasoning_tokens: 0 }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const modelWithCaching = new ChatOpenAI({\n",
        "  model: \"gpt-4o-mini-2024-07-18\",\n",
        "});\n",
        "\n",
        "// CACHED_TEXT is some string longer than 1024 tokens\n",
        "const LONG_TEXT = `You are a pirate. Always respond in pirate dialect.\n",
        "\n",
        "Use the following as context when answering questions:\n",
        "\n",
        "${CACHED_TEXT}`;\n",
        "\n",
        "const longMessages = [\n",
        "  {\n",
        "    role: \"system\",\n",
        "    content: LONG_TEXT,\n",
        "  },\n",
        "  {\n",
        "    role: \"user\",\n",
        "    content: \"What types of messages are supported in LangChain?\",\n",
        "  },\n",
        "];\n",
        "\n",
        "const originalRes = await modelWithCaching.invoke(longMessages);\n",
        "\n",
        "console.log(\"USAGE:\", originalRes.response_metadata.usage);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "76c8005e",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "USAGE: {\n",
            "  prompt_tokens: 2624,\n",
            "  completion_tokens: 272,\n",
            "  total_tokens: 2896,\n",
            "  prompt_tokens_details: { cached_tokens: 2432 },\n",
            "  completion_tokens_details: { reasoning_tokens: 0 }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "const resWitCaching = await modelWithCaching.invoke(longMessages);\n",
        "\n",
        "console.log(\"USAGE:\", resWitCaching.response_metadata.usage);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f755a0b3",
      "metadata": {},
      "source": [
        "## Predicted output\n",
        "\n",
        "Some OpenAI models (such as their `gpt-4o` and `gpt-4o-mini` series) support [Predicted Outputs](https://platform.openai.com/docs/guides/latency-optimization#use-predicted-outputs), which allow you to pass in a known portion of the LLM's expected output ahead of time to reduce latency. This is useful for cases such as editing text or code, where only a small part of the model's output will change.\n",
        "\n",
        "Here's an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "4d5a5582",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "AIMessage {\n",
            "  \"id\": \"chatcmpl-AQLyQKnazr7lEV7ejLTo1UqhzHDBl\",\n",
            "  \"content\": \"/// <summary>\\n/// Represents a user with a first name, last name, and email.\\n/// </summary>\\npublic class User\\n{\\n/// <summary>\\n/// Gets or sets the user's first name.\\n/// </summary>\\npublic string FirstName { get; set; }\\n\\n/// <summary>\\n/// Gets or sets the user's last name.\\n/// </summary>\\npublic string LastName { get; set; }\\n\\n/// <summary>\\n/// Gets or sets the user's email.\\n/// </summary>\\npublic string Email { get; set; }\\n}\",\n",
            "  \"additional_kwargs\": {},\n",
            "  \"response_metadata\": {\n",
            "    \"tokenUsage\": {\n",
            "      \"promptTokens\": 148,\n",
            "      \"completionTokens\": 217,\n",
            "      \"totalTokens\": 365\n",
            "    },\n",
            "    \"finish_reason\": \"stop\",\n",
            "    \"usage\": {\n",
            "      \"prompt_tokens\": 148,\n",
            "      \"completion_tokens\": 217,\n",
            "      \"total_tokens\": 365,\n",
            "      \"prompt_tokens_details\": {\n",
            "        \"cached_tokens\": 0\n",
            "      },\n",
            "      \"completion_tokens_details\": {\n",
            "        \"reasoning_tokens\": 0,\n",
            "        \"accepted_prediction_tokens\": 36,\n",
            "        \"rejected_prediction_tokens\": 116\n",
            "      }\n",
            "    },\n",
            "    \"system_fingerprint\": \"fp_0ba0d124f1\"\n",
            "  },\n",
            "  \"tool_calls\": [],\n",
            "  \"invalid_tool_calls\": [],\n",
            "  \"usage_metadata\": {\n",
            "    \"output_tokens\": 217,\n",
            "    \"input_tokens\": 148,\n",
            "    \"total_tokens\": 365,\n",
            "    \"input_token_details\": {\n",
            "      \"cache_read\": 0\n",
            "    },\n",
            "    \"output_token_details\": {\n",
            "      \"reasoning\": 0\n",
            "    }\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const modelWithPredictions = new ChatOpenAI({\n",
        "  model: \"gpt-4o-mini\",\n",
        "});\n",
        "\n",
        "const codeSample = `\n",
        "/// <summary>\n",
        "/// Represents a user with a first name, last name, and username.\n",
        "/// </summary>\n",
        "public class User\n",
        "{\n",
        "/// <summary>\n",
        "/// Gets or sets the user's first name.\n",
        "/// </summary>\n",
        "public string FirstName { get; set; }\n",
        "\n",
        "/// <summary>\n",
        "/// Gets or sets the user's last name.\n",
        "/// </summary>\n",
        "public string LastName { get; set; }\n",
        "\n",
        "/// <summary>\n",
        "/// Gets or sets the user's username.\n",
        "/// </summary>\n",
        "public string Username { get; set; }\n",
        "}\n",
        "`;\n",
        "\n",
        "// Can also be attached ahead of time\n",
        "// using `model.bind({ prediction: {...} })`;\n",
        "await modelWithPredictions.invoke(\n",
        "  [\n",
        "    {\n",
        "      role: \"user\",\n",
        "      content:\n",
        "        \"Replace the Username property with an Email property. Respond only with code, and with no markdown formatting.\",\n",
        "    },\n",
        "    {\n",
        "      role: \"user\",\n",
        "      content: codeSample,\n",
        "    },\n",
        "  ],\n",
        "  {\n",
        "    prediction: {\n",
        "      type: \"content\",\n",
        "      content: codeSample,\n",
        "    },\n",
        "  }\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "81f901e4",
      "metadata": {},
      "source": [
        "Note that currently predictions are billed as additional tokens and will increase your usage and costs in exchange for this reduced latency."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cc8b3c94",
      "metadata": {},
      "source": [
        "## Audio output\n",
        "\n",
        "Some OpenAI models (such as `gpt-4o-audio-preview`) support generating audio output. This example shows how to use that feature:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "b4d579b7",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  id: 'audio_67129e9466f48190be70372922464162',\n",
            "  data: 'UklGRgZ4BABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAATElTVBoAAABJTkZPSVNGVA4AAABMYXZmNTguMjkuMTAwAGRhdGHA',\n",
            "  expires_at: 1729277092,\n",
            "  transcript: \"Why did the cat sit on the computer's keyboard? Because it wanted to keep an eye on the mouse!\"\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const modelWithAudioOutput = new ChatOpenAI({\n",
        "  model: \"gpt-4o-audio-preview\",\n",
        "  // You may also pass these fields to `.bind` as a call argument.\n",
        "  modalities: [\"text\", \"audio\"], // Specifies that the model should output audio.\n",
        "  audio: {\n",
        "    voice: \"alloy\",\n",
        "    format: \"wav\",\n",
        "  },\n",
        "});\n",
        "\n",
        "const audioOutputResult = await modelWithAudioOutput.invoke(\"Tell me a joke about cats.\");\n",
        "const castAudioContent = audioOutputResult.additional_kwargs.audio as Record<string, any>;\n",
        "\n",
        "console.log({\n",
        "  ...castAudioContent,\n",
        "  data: castAudioContent.data.slice(0, 100) // Sliced for brevity\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bfea3608",
      "metadata": {},
      "source": [
        "We see that the audio data is returned inside the `data` field. We are also provided an `expires_at` date field. This field represents the date the audio response will no longer be accessible on the server for use in multi-turn conversations.\n",
        "\n",
        "### Streaming Audio Output\n",
        "\n",
        "OpenAI also supports streaming audio output. Here's an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "0fa68183",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  id: 'audio_67129e976ce081908103ba4947399a3eaudio_67129e976ce081908103ba4947399a3e',\n",
            "  transcript: 'Why was the cat sitting on the computer? Because it wanted to keep an eye on the mouse!',\n",
            "  index: 0,\n",
            "  data: 'CgAGAAIADAAAAA0AAwAJAAcACQAJAAQABQABAAgABQAPAAAACAADAAUAAwD8/wUA+f8MAPv/CAD7/wUA///8/wUA/f8DAPj/AgD6',\n",
            "  expires_at: 1729277096\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "import { AIMessageChunk } from \"@langchain/core/messages\";\n",
        "import { concat } from \"@langchain/core/utils/stream\"\n",
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const modelWithStreamingAudioOutput = new ChatOpenAI({\n",
        "  model: \"gpt-4o-audio-preview\",\n",
        "  modalities: [\"text\", \"audio\"],\n",
        "  audio: {\n",
        "    voice: \"alloy\",\n",
        "    format: \"pcm16\", // Format must be `pcm16` for streaming\n",
        "  },\n",
        "});\n",
        "\n",
        "const audioOutputStream = await modelWithStreamingAudioOutput.stream(\"Tell me a joke about cats.\");\n",
        "let finalAudioOutputMsg: AIMessageChunk | undefined;\n",
        "for await (const chunk of audioOutputStream) {\n",
        "  finalAudioOutputMsg = finalAudioOutputMsg ? concat(finalAudioOutputMsg, chunk) : chunk;\n",
        "}\n",
        "const castStreamedAudioContent = finalAudioOutputMsg?.additional_kwargs.audio as Record<string, any>;\n",
        "\n",
        "console.log({\n",
        "  ...castStreamedAudioContent,\n",
        "  data: castStreamedAudioContent.data.slice(0, 100) // Sliced for brevity\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e8b84aac",
      "metadata": {},
      "source": [
        "### Audio input\n",
        "\n",
        "These models also support passing audio as input. For this, you must specify `input_audio` fields as seen below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "1a69dad8",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "That's a great joke! It's always fun to imagine why cats do the funny things they do. Keeping an eye on the \"mouse\" is a creatively punny way to describe it!\n"
          ]
        }
      ],
      "source": [
        "import { HumanMessage } from \"@langchain/core/messages\";\n",
        "\n",
        "const userInput = new HumanMessage({\n",
        "  content: [{\n",
        "    type: \"input_audio\",\n",
        "    input_audio: {\n",
        "      data: castAudioContent.data, // Re-use the base64 data from the first example\n",
        "      format: \"wav\",\n",
        "    },\n",
        "  }]\n",
        "})\n",
        "\n",
        "// Re-use the same model instance\n",
        "const userInputAudioRes = await modelWithAudioOutput.invoke([userInput]);\n",
        "\n",
        "console.log((userInputAudioRes.additional_kwargs.audio as Record<string, any>).transcript);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",
      "metadata": {},
      "source": [
        "## API reference\n",
        "\n",
        "For detailed documentation of all ChatOpenAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "TypeScript",
      "language": "typescript",
      "name": "tslab"
    },
    "language_info": {
      "codemirror_mode": {
        "mode": "typescript",
        "name": "javascript",
        "typescript": true
      },
      "file_extension": ".ts",
      "mimetype": "text/typescript",
      "name": "typescript",
      "version": "3.7.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
